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5 A METHOD AND A SYSTEM FOR PROGRAMMING AN IN- 
DUSTRIAL ROBOT 

FIELD OF THE INVENTION 

10 The present invention relates to a method in connection with 
programming of an industrial robot, comprising teaching the ro- 
bot a path which has a number of waypoints located on or in the 
vicinity of an object to be processed by the robot. The process- 
ing of the object could be any type of surface processing appli- 

15 cation carried out by an industrial robot, such as painting, glaz- 
ing, sanding, polishing, grinding, welding, picking and placing, 
and assembling. 

The present invention also relates to a computer program prod- 
20 uct directly loadable into the internal memory of a computer, 
comprising software code portions for performing the steps of 
the method according to the invention, when said product is run 
on a computer. 

25 The present invention also relates to a computer readable me- 
dium having a program recorded thereon, where the program is 
to make a computer perform the steps of the method according 
to the invention when said program is run on the computer. 

30 The present invention also relates to the use of a method ac- 
cording to the invention for a paint application. 

The present invention further relates to a system for use in con- 
nection with programming of an industrial robot, the system 
35 comprising information obtaining means, obtaining information 
about the waypoints of the path in relation to the object, a stor- 
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age unit, for storing the obtained information, a graphics gen- 
erator, generating a graphical representation, and a display 
member displaying a view comprising the object and the graphi- 
cal representation generated by the graphics generator. 

5 

PRIOR ART 

Robots are often used for processing the surface of an object. 
Existing practice for programming a robot involves teaching the 

10 robot a sequence of waypoints. The waypoints define the path, 
which the robot shall follow during the processing of the object. 
A waypoint comprises three-dimensional position and three- 
dimensional orientation information. The robot is taught how to 
perform the task by being guided through the various waypoints 

15 along the desired operating path during the programming. These 
waypoints are stored as instructions into a memory in the robot 
control unit. During operation of the robot, the program instruc- 
tions are executed, thereby making the robot operate as de- 
sired. 

20 

Generally, the robot is programmed to perform a task by an hu- 
man operator who manually jogs the robot to the desired posi- 
tions along the path using a movable programming unit, a so- 
called teach pendant. An industrial robot usually carries an end- 

25 effector comprising a tool for performing the processing, for in- 
stance a welding tool or a painting tool. During programming of 
the path, the operator has to position and orient the end-effector 
in each waypoint along the path. Alternatively, if a 3D CAD 
model of the object exists, a person with a computer scientist 

30 background teaches the waypoints in a robot simulation system, 
so called off-line programming. Existing methods based on the 
CAD approach include visual feedback in an entirely virtual 
world, i.e. a virtual representation of both the real world and the 
object. 
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Either way, the teaching process is time consuming, trouble- 
some, error prone, and in almost all cases requires several it- 
erations before the program is acceptable. The CAD approach is 
costly, it requires a 3D CAD model of the object and it is not al- 
5 ways intuitive to use. The complexity of the CAD-system re- 
quires the operator who is programming the robot to have knowl- 
edge about computer science. Such a person usually has little 
or no knowledge about the process. In the case of manual 
teaching, the control of the process is often difficult to optimize, 

10 but it has the advantage of indirectly utilizing implicit process 
knowledge of the operator. Another disadvantage with the man- 
ual teaching method is that it does not include any visual feed- 
back to the operator, visualizing what has been programmed. 
The operator must use the trial and error method until the pro- 

15 gram is acceptable. For example, when the robot is to be taught 
how to paint an object and the entire surface of the object must 
be covered with color, it is impossible for the operator, without 
running the program afterwards, to see if he has missed some 
part of the surface. Thus, a number of iterations are required 

20 before the quality of the processing, in this example the paint- 
ing, is satisfactory. 

From the Japanese patent JP10011122 an improved method for 
teaching an industrial robot is known. The method includes pre- 

25 senting a visual feedback to the operator of the response result- 
ing from an operation before the robot is operated. The visual 
feedback is represented together with the real object. The 
method comprises: measuring the current state of the robot and 
its environment by a CCD camera, the operator inputs robot op- 

30 eration information by the teaching pendant, estimating a re- 
sponse after the operation of the robot based on the inputted 
information, converting the estimated operation into information 
to be displayed as a picture, preparing picture information re- 
lated to the response of the robot based on the information re- 

35 lated to the camera and the estimated information, synthesizing 
the estimated response based on the picture information meas- 
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ured by the CCD camera and the picture information related to 
the response of the robot, and displaying a view comprising the 
robot, its environment and the estimated response of the robot. 
Thus, this method shows the next operation to be performed by 
5 the robot, but it does not provide any visual feedback to the op- 
erator regarding what he has been programmed. 

SUMMARY OF THE INVENTION 

10 The object of the present invention is to provide an improved 
industrial robot programming solution, which reduces the teach- 
ing time for the teaching process and increases the quality of 
the processing result. 

15 According to one aspect of the invention, the object is achieved 
by means of the initially defined method, comprising: obtaining 
information about the position of the waypoints in relation to the 
object, storing the information about the position of the way- 
points, simulating the robot path based on the received informa- 

20 tion about the waypoints and a model of the robot, generating a 
graphical representation of the robot path based on the simu- 
lated robot path, and displaying a view comprising the object 
and said graphical representation of the robot path projected on 
the object. This method improves the teaching process by visu- 

25 alizing the robot path in relation to the real object to be pro- 
cessed. 

Obviously, it is not efficient if the robot moves the tool in 
straight lines between the waypoints specified by the operator. 

30 In order to achieve an efficient robot program, effecting the tool 
to pass through the specified waypoint at required speed and di- 
rection, a robot path has to be generated. The robot path is ge- 
nerated from a path simulator, simulating the robot path based 
on specified waypoints and a model of the robot. Computer gen- 

35 erated information, representing the generated robot path, is 
displayed projected on the real object or on an image of the real 
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object. Thus, it is possible for the operator to get extended vis- 
ual feedback while making a new robot program related to a 
specific object. It is no longer necessary to run the robot pro- 
gram to view the result of the programming. Thereby, the num- 
5 ber of iterations needed and thus the teaching time is reduced. 
The method is particularly advantageously during off-line pro- 
gramming. Another advantage of the method is that no 3D CAD 
model of the object is needed, since the generated graphic is 
projected on the real object or on an image of the real object. 
10 The programming time will be significantly reduced since the 
operator will be able to make a robot program at a higher quality 
level during the first iteration compared to today's methods. 

According to a further embodiment of the invention, the method 
15 comprises obtaining information about the position of a display 
member in relation to the object and displaying said view in de- 
pendence of the position of the display member in relation to the 
object. Preferably, the orientation of the display member is also 
obtained and said view is displayed in dependence of the posi- 
20 tion and orientation of the display member. Thus the view dis- 
played on the display member depends on its position and orien- 
tation in relation to the object. Thus, the operator can walk 
around the real object and se the result of what he has been 
teaching on the real object from different viewpoints. 

25 

According to a further embodiment of the invention, the method 
comprises: obtaining information about tool orientations in the 
waypoints, and generating a graphical representation of the tool 
orientations in the waypoints. During programming of the robot, 
30 the operator also needs to teach the robot how to orientate the 
tool. The programming is further simplified if the orientations of 
the tool in the waypoints are visualized to the operator. 

According to a further embodiment of the invention, the method 
35 comprises: obtaining information about the process to be per- 
formed by the robot in connection with the robot path, simulating 
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the result of the process based upon the obtained information 
about the waypoints, the obtained information about the process 
and a model of the process, generating a graphical representa- 
tion of the simulated result of the process, and displaying a view 
5 showing the graphical representation of the simulated result of 
the process projected on the object. The information received is 
for example information about the tool configuration, such as the 
brush to be used in a painting process. Thereby, it is possible 
for the operator to estimate if the tool configuration chosen 

10 achieves the desired result. The final result of the process per- 
formed by the robot is presented to the operator in the real work 
environment. In the painting example, the result of the painting 
on the surface of the object is displayed. The method visualizes 
the outcome of the programming process to the operator, with- 

15 out the need of running the robot program or start the process. 
Preferably, it is possible for the operator to choose between 
visualizing the robot path, the result of the process, or both the 
robot path and the result of the process at the same time. 

20 The visualization of the robot path and/or the outcome from the 
process may either be visualized, while the operator teaches a 
new robot program, or the robot path and/or the outcome from 
the process may be visualized as a function of time after the op- 
erator has completed the teaching, but before the robot program 

25 code is generated and downloaded to the robot controller. In 
both cases the operator is able to investigate the simulated re- 
sult of the robot program without having to execute the new ro- 
bot program in reality. 

30 During the visualization of the robot processing as a function of 
time, the method supports stopping, pausing, restarting, and 
play backwards the simulation. 

According to a further embodiment of the invention, the method 
35 comprises: obtaining information about which tool to be used to 
perform the process and about the orientation of the tool in the 
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waypoints, simulating the tool performing the process, generat- 
ing a graphical representation of the tool performing the process 
along the robot path based upon the obtained information about 
the position of the waypoints and the orientation of the tool, and 
5 displaying a view showing the tool moving along the robot path 
performing the process, based on the generated graphical rep- 
resentation of the simulation of the tool performing the process. 
For example, if the process is painting, a virtual paint-gun is 
displayed moving along the robot path performing the paint 
10 procedure and the paint being applied onto the object is shown. 
This visualization of the tool performing the process helps the 
operator to evaluate the taught robot program. 

According to a further embodiment of the invention the view of 
15 the robot path and/or the result of the process is displayed as a 
function of time and the view is displayed proportional to the 
real-time. Real-time is the actual time it takes for the robot to 
move through the path performing the process. In a preferred 
embodiment the method further comprises: receiving information 
20 about a desired speed of the displaying of the view, and display- 
ing the view in accordance with the desired speed. Preferably, 
the speed is altered in relation to the real-time. Sometimes the 
operator finds the real-time speed to be too fast, and thus 
wishes to slow down the speed. This embodiment of the inven- 
25 tion makes it possible for the operator to alter the speed of the 
display. 

According to a further embodiment of the invention the method 
comprises generating different graphical representations de- 

30 pending on whether the result of the process is displayed from a 
front side or a backside in relation to the object. Thus it is easy 
for the operator to distinguish between back and front. For ex- 
ample, a rendered paint stroke is shown with a front side and a 
backside with the backside rendered with transparent graphics. 

35 That makes it possible to turn off the graphical process informa- 
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tion, for example the generation of paint stroke, when the back- 
side of the paint stroke in shown. 

According to a further embodiment of the invention the method 
5 comprises simulating the quality of the result of the process 
based on one or a plurality of predefined quality parameters and 
the model of the process, generating a graphical representation 
of the quality of the result of the process, and displaying the 
view based on the generated graphical representation of the 

10 simulation of the quality of the result of the process. Preferably, 
the method further comprises estimating whether the quality of 
the result does not meet one or several quality requirements, 
based on said simulation of the quality of the result of the proc- 
ess, and generating said graphical representation with a visual 

15 feedback to the operator where the process quality is estimated 
not to meet the quality requirements. In the paint process, the 
quality parameters for instance are overlap between paint 
strokes, parts of the object not covered by the paint, the paint 
thickness, the paint width, and sag of the paint. Thus, if the 

20 paint is estimated to be sagging or not covering the object, this 
will be shown by the generated graphics. This embodiment 
makes it possible for the operator to generate robot programs 
with high quality, without the need of executing the actual proc- 
ess. 

25 

According to a further embodiment of the invention the method 
comprises determining whether a point on the robot path is 
within the working range of the robot and notifying the operator 
if the point is outside the working range. Whether a point is 

30 within or outside the working range of the robot is determined 
based on a model of the robot. When the operator is specifying 
new waypoints or editing stored waypoints it is essential that the 
waypoints are within the working range of the robot. If a speci- 
fied waypoint is outside the working range the operator is noti- 

35 fied. It is also possible to check if any of the points on the robot 
path generated by the simulator is outside the working range. 
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When a point cases a singularity of the robot the point is outside 
the working rage. 

According to a further embodiment of the invention the method 
5 comprises obtaining an image of the object, registering the gen- 
erated graphical representation to the image of the object to 
provide a composite augmented reality image and displaying 
said view based on the composite augmented reality image. 
Preferably the image of the object is obtained by means of a 

10 camera. An augmented reality system merges computer- 
generated graphics of objects with the user's space in the real 
world. In this embodiment, the computer generated graphics of 
the waypoints and paths are merged with the image of the real 
object. Alternatively, the computer-generated graphics of way- 

15 points and paths is displayed on see-through glasses worn by 
the operator. The generated graphics are projected on the 
glasses so that the operator can see the waypoints and paths 
projected in relation to the real world and the object. In order to 
perform a correct graphics registration, a tracking system is 

20 used to obtain the pose of the projection surface of the display 
device in relation to the object coordinate frame. 

According to a further embodiment of the invention the method 
comprises obtaining information about the position of a pointing 

25 member pointing at points on or in the vicinity of the object, de- 
termining the position of the points in relation to the object co- 
ordinate frame based upon said obtained information, and stor- 
ing the point being presently pointed out by the pointing member 
as a waypoint upon receiving a recording signal. Thus, the op- 

30 erator can specify the waypoints by walking around the object 
and pointing out the waypoints using the pointing member. The 
operator does not need any particular computer knowledge and 
he uses his knowledge and experience about the process during 
teaching of the robot. 

35 
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According to a further aspect of the invention, the object is 
achieved by a computer program product directly loadable into 
the internal memory of a computer, comprising software code 
portions for performing the steps of the method according to the 
5 invention, when said product is run on a computer. The com- 
puter program product is provided either on a computer readable 
medium or through a network such as the Internet. 

According to another aspect of the invention, the object is 
10 achieved by a computer readable medium having a program re- 
corded thereon, where the program is to make a computer per- 
form the steps of the method according to the invention, when 
said program is run on the computer. 

15 According to still another aspect of the invention, the object is 
achieved by the initially defined system characterized in that it 
comprises a simulation unit, simulating the robot path based on 
the obtained information about the waypoints and a model of the 
robot, that the graphics generator, is adapted for generating a 

20 graphical representation of the simulated robot path, preferably 
along with process related information, and that the display 
member, is adapted for displaying a view showing said graphical 
representation of the simulated robot path projected on the ob- 
ject. 

25 

According to an embodiment of the invention the system com- 
prises a handheld display device comprising the display member 
and a camera. Preferably the handheld display device is ar- 
ranged so that the user seems to look directly through the dis- 
30 play. A handheld display device is useful when a user needs to 
see the computer-generated information from time to time. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The invention will now be explained more closely by the descrip- 
tion of different embodiments thereof and with reference to the 
5 appended figures. 

Fig. 1 shows a system for use in connection with programming 
of an industrial robot, according to an embodiment of the 
invention. 

10 

Fig. 2 shows a block diagram of the system according to an em- 
bodiment of the invention. 



Fig. 3 shows examples of an augmented reality view showing a 
15 graphical representation of the robot path projected on a 

real object and path information. 

Figs. 4a and 4b show flow charts of a method according to the 
invention for teaching an industrial robot a path according 
20 to an embodiment of the invention. 

Fig. 5 shows an embodiment of the invention comprising a 
handheld display device. 

25 DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
OF THE INVENTION 



Fig. 1 illustrates a system for use in connection with program- 
ming of an industrial robot according to the present invention. 

30 The system comprises a pointing member 1 utilized by a human 
operator 2 for teaching the robot how to process an object 3. 
The pointing member can either be the processing tool itself, or 
a device particularly adapted for the purpose of teaching the 
processing of the object. The teaching comprises teaching a 

35 number of waypoints given in a specific sequence and located 
on or in the close vicinity of the object 3. Each waypoint com- 
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prises the position of a point on or in a close vicinity of the ob- 
ject, the orientation of a processing tool, such as a paintbrush, 
in the point, and data related to the process. 

5 The operator holds the pointing member 1 in his hand and points 
at the point he wants to include in the path and orientates the 
pointing member as he wishes the processing tool to be oriented 
in the point. The operator records a point by activation of an ac- 
tivation member, which generates a recording signal. Upon re- 

10 ceiving the recording signal the system stores the point as a 
waypoint. The pointing member 1 is provided with at least one 
marker 5, called a pointing marker, attached thereto. The point- 
ing marker 5 is used for determining the position and orientation 
of the pointing member 1 in relation to the object. Alternatively, 

15 the pointing device may generate positions and orientations in a 
number of different ways, e.g. with inertial sensors, gyros, mag- 
netic trackers, laser-based tracking systems, and ultrasonic 
trackers producing positions and orientations of the pointing de- 
vice in relation to a fixed coordinate system. 

20 

The system is based on image recognition and requires at least 
one reference marker 6, which is fixed in relation to the object. 
The function of the reference marker 6 is to make up a reference 
point for image recognition algorithms, and a reference point to 

25 which computer generated graphics should be related. The sys- 
tem further comprises a camera unit 8. The camera unit 8 com- 
prises an optical camera and an image grabber feeding the sys- 
tem with real-time image data. The camera can either be fixed in 
the space or moved around. For instance the camera can be 

30 attached to the operator's head or elsewhere on his body. In this 
embodiment the camera is attached to a par of glasses worn by 
the operator. All types of cameras can be used, but the demand 
for accuracy of position and orientation determines the type of 
camera. For example, a web camera, a video camera, or a CCD 

35 camera could be used. In another embodiment, more than one 
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camera can be used, for example two cameras could be used 
arranged so as to provide stereoscopic images. 

The camera provides either an analogue or a digital video sig- 
5 nal, which is transferred to a computer 10 including a processor. 
The processor of the computer 10 is utilized for running the im- 
age recognition algorithms for determining the position and ori- 
entation of the pointing member 1 in relation to the object, for 
running algorithms generating a graphical representation of way- 
10 points pointed out by the operator, for combining the generated 
graphics of the waypoints with the image provided by the cam- 
era to provide a composite augmented reality image and for 
running the simulation of the robot path and the result of the 
process. 

15 

The system further comprises a graphical display member 12 on 
which computer generated graphics is displayed to the operator. 
The graphical display member 12 visualizes the view of the cam- 
era combined with the computer-generated graphics. In this 

20 embodiment, the operator wears a head-mounted unit including a 
head-mounted graphical display 12 with the camera attached to the 
display. The head-mounted unit is a pair of wearable glasses on 
which the computer-generated information is projected in rela- 
tion to the operator's position in the room. The view displayed 

25 depends on the position of the display member in relation to the 
object. The position of the display in relation to the object is de- 
termined by image recognition based on the image from the 
camera mounted on the display and markers (not shown) pro- 
vided on the object, or some other tracking system, e.g. with in- 

30 ertial sensors, gyros, magnetic trackers, laser-based tracking 
systems, ultrasonic trackers, or vision-based feature tracking. 
Alternatively, a head-mounted display is used. In this embodi- 
ment, the camera is mounted to the display device, producing 
the same view as the operator would see. The computer-gener- 

35 ated information is then registered with the camera image before 
displayed to the operator. In another embodiment, the display 
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member is for example a handheld screen, such as a Personal 
Digital Assistant (PDA) screen, or a laptop screen. In the case 
of a PDA, the operator will hold the PDA in his hand while pro- 
gramming and the outcome of the programming will be visual- 
5 ized on the PDA screen. The display member could either give a 
single view or a stereoscopic representation of the graphics. 

The pointing member 1 comprises a number of interaction 
means, such as push-buttons, which are adapted for interaction 
between the operator and the system. One of the interaction 
means comprises an activating member, which upon activation 
generates a recording signal for recording the point as a way- 
point. A second interaction means is used for indicating whether 
the processing should be on or off between the waypoints. For 
example, if the process is painting, the interaction means indi- 
cates whether the paint is on or off. The system is provided with 
a third interaction means for input of information related to the 
process to the system. For example, in a paint application, it is 
possible to select one or several options for the paintbrush sto- 
red in the robot controller. For other applications, for example 
polishing, the options relates to the configuration of the polish- 
ing brush. 

The relationship between the marker 5 and the tip 4 of the point- 
25 ing member 1 must be known, so that the position and orienta- 
tion of the tip can be determined. The image recognition algo- 
rithms recognize the pointing marker 5. The position of the tip of 
the pointing member and the orientation of the body of the point- 
ing member in the reference marker coordinate system is deter- 
30 mined by the recognition of the markers 5, 6. The recognition is 
based on the size and angle of the pattern on the marker in rela- 
tion to the known pattern learned by the image recognition algo- 
rithms. The position is given in 3D as [x, y, z]. The orientation is 
given in 3D, for example as a rotation matrix [a 3x3 matrix]. 
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While teaching the robot, the operator records waypoints to be 
passed through during the processing of the object and he also 
records process related information in connection with the way- 
points by using the interaction means. In addition to the posi- 
5 tion, the orientation of the pointing member is recorded and the 
recorded orientation represents the orientation of the robot tool. 
Hence, the recorded sequence of waypoints creates the basis 
for the robot path in the robot program. 

Fig. 2 shows a block diagram of a system according to an 
embodiment of the invention. The system comprises a tracking 
unit 14, which determines the positions and orientations of the 
way points taught by the operator. The tracking unit allows pose 
information (3DOF position and 3DOF orientation) to be speci- 
fied in relation to a fixed object coordinate system. In this 
embodiment the position and orientation is determined by means 
of image recognition. Examples of other tracking units are 
ultrasonic tracking systems, magnetic tracking systems, tracking 
systems based upon inertial navigation, which uses 
accelerometers and gyros, mechanical arms, vision-based and 
optical systems, and hybrid systems combining the previously 
mentioned technologies. In this embodiment, input data to the 
tracking unit 14 is video signals from the camera unit 8 and in- 
formation from the activating members of the pointing member 
1, such as a record signal and process related information. Al- 
ternatively, the waypoints are loaded into the system from an 
external system, e.g. a CAD application, or from a database of 
stored waypoints and process information. 

30 The system further comprises a point generator 15, which gen- 
erates a sequence of points on the robot path with associated 
process related information based upon the waypoints specified 
from the tracking system. For a paint application, point data will 
typically be position and orientation of the brush, fan descrip- 

35 tion, paint color, paint on/off, etc. The point generator will mod- 
ify the waypoints specified by the operator depending on the 
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process related information, such as brush number, and configu- 
ration parameters. Input to the point generator is waypoints with 
associated process information provided by the tracking unit 14. 
Output from the point generator 15 is a sequence of modified 
5 waypoints with process specific information. The point generator 
also adds new waypoints to the sequence of waypoints already 
specified by the operator. The sequence of waypoints generated 
by the point generator is stored in a storage unit 16, which also 
is used for storing system information. 

10 

A robot path simulator 18 simulates the actual robot path based 
on the list of waypoints and process information generated by 
the point generator 15. The robot path simulator comprises a 
model of the robot and the control system of the robot. The ro- 

15 bot path simulator generates the actual robot path based on the 
waypoints, the process information, and the model of the robot 
and the robot control system. The generated robot path is used 
both for visualizing the robot movements in real time, i.e. as a 
function of time, and for visualizing a trace of the actual robot 

20 path. Input to the robot path simulator is the sequence of way- 
points from the point generator and robot configuration data. 
The output from the robot path simulator is the actual robot path 
comprising position and orientation of the tool. 

25 A reachability unit 20 checks whether a point is reachable for 
the robot end effector. The reachability unit 20 checks whether a 
point is within the working range of the robot and if the point is a 
singularity. When the operator specifies new waypoints or edits 
stored waypoints, it is essential that the waypoints and the path 

30 between the waypoints are reachable. Specifying a waypoint or 
a path which is not reachable, results in a visual message to the 
operator, e.g. by rendering the infeasible point or path with a 
dedicated color, pattern, or animation. In one embodiment of the 
system, the reachability of the waypoints specified by the opera- 

35 tor is checked. In another embodiment of the system, the reach- 
ability of the actual robot path generated from the path simulator 



17 



18 is checked. This is advantageous since the specified way- 
points may be feasible but the generated path between the way- 
points might not be feasible, e.g. because of singularities or 
reachability constraints. The reachability is calculated based on 
5 a model of the robot and the robot controller, which model is the 
same as the one used by the robot path simulator 18. The calcu- 
lations are possible thanks to knowledge about the position of 
the robot in relation to the object coordinate system. Singulari- 
ties are possible to detect thanks to the model of the robot con- 
10 troller. 

A process simulator 22 simulates the result from the process 
based on the waypoints, the process information from the point 
generator 15, the actual robot path, and a model of the process. 

15 Thus, the simulation depends on which process to be simulated. 
The process simulator uses a model of the process to be simu- 
lated. The process simulator may comprise more than one proc- 
ess model. For example it may comprise one model for a paint 
application, one model for a mill application and one model for a 

20 welding application. As an example, the model for the paint ap- 
plication comprises a combination of a mathematical model and 
a neural network model, where the mathematical model simu- 
lates the paint application, and the neural network simulates the 
paint result at the object surface. For instance, the width of the 

25 simulated paint-stroke is calculated from the brush number and 
the distance from the tool center point (TCP) to the object. The 
distance from the TCP to the object is fixed and is specified by 
the pointing member. 

30 The process simulator 22 also calculates the process quality 
based on one or more predefined quality parameters. For paint 
applications, examples of relevant quality parameters are over- 
lap between paint strokes, parts of the object not covered by the 
paint, paint thickness, paint width, and dripping of the paint. The 

35 quality simulation is based on the previous mentioned model of 
the process. For paint applications, a model of the paint process 
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may be parameterized from a set of samples taken with different 
system and environmental parameters, such as brushes, paint 
type, humidity, and temperature. Based upon these data, a 
model of the process is generated and the paint thickness 
5 and/or stroke width is calculated. In order to calculate these 
properties, a model of the object may be needed. Such an object 
model can be provided in different ways. For example, by use of 
a CAD model, an optical system generating a 3D model of the 
object by automatic inspection or by using the tracking unit to 
10 specify points on the object and from this data, generate a sim- 
plified 3D model of the object. The process result should take 
into account if paint strokes overlap. If there exits a 3D model of 
the object, the width of the paint strokes is displayed as it would 
be on the real object. 

15 

Based on the simulation of the quality of the process, an estima- 
tion of whether the quality of the result does or does not meet 
one or several predefined quality requirements is done. The 
quality requirements are, for example, a minimum and a maxi- 

20 mum value for the thickness of the paint and an interval for the 
overlap between two paint strokes. If the estimation shows that 
the process quality does not to meet the quality requirements a 
graphical representation with a visual warning to the operator is 
generated. For paint applications, the paint thickness is ren- 

25 dered in different ways, for example by a different color or a dif- 
ferent pattern depending on whether the process result does or 
does not meet the quality requirement, or the simulated paint 
thickness may be illustrated continuously along the paint strokes 
by a graphical property, such as color, color intensity, or color 

30 saturation. 

A graphics generator 23 generates a 3D graphical representa- 
tion of the virtual information to be displayed based on informa- 
tion about the robot path from the robot path simulator and in- 
35 formation about the process obtained from the process simulator 
22. The graphics generator 23 generates a graphical represen- 
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tation of the simulated robot path and a graphical representation 
of the simulated result of the process. The graphics generator 
23 also generates a graphical representation of the tool perform- 
ing the process along the simulated robot path. The graphical 
5 representations are descriptions of the 3D graphical elements, 
specified in relation to the object coordinate frame. The system 
further comprises a registering unit 25, registering the generated 
graphical representation to the image of the object received 
from the camera unit 8, and providing a composite augmented 

10 reality image. The input to the registering unit 25 is the graphi- 
cal representation from the graphics generator 23, video signals 
from the camera unit 8, and the position of the display device 
from the object-tracking unit 24. The output from the registering 
unit 25 is a composite augmented reality image, which is dis- 

15 played on the display member 12. 

The graphics generator 23, the registering unit 25, and the cam- 
era unit 8 form a graphics rendering system, which is configured 
for a video see-through display, i.e. computer generated graph- 

20 ics are combined with a captured real world image. Alternatively, 
the graphics rendering system could be configured for an optical 
see-through display, for which only computer-generated graph- 
ics are rendered. Further, the graphics rendering system is con- 
figured for either a single-camera system or for a stereoscopic 

25 camera system, having two cameras capturing real world images 
from slightly different positions, giving a tree-dimensional repre- 
sentation of the real world. The graphics rendered by the graph- 
ics rendering system is displayed to the operator on the graphi- 
cal display 12. Thus, the system projects computer generated 

30 information, which simulates the specific process, directly onto 
and/or in relation to the real object to be processed. This infor- 
mation represents the robot path and process related informa- 
tion taught by the operator. 

35 The system comprises an object-tracking unit 24, delivering in- 
formation about the positional relation between the object and 
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the display member 12. This information is input to the register- 
ing unit 25. Preferably, the object tracking is based on the same 
tracking method as the waypoint-tracking unit and uses the 
same tracking device. In this embodiment the object tracking is 
5 based on image recognition of the image received from the 
camera unit 8. The object-tracking unit could also be any of the 
before mentioned examples of tracking systems. An object- 
tracking unit is needed if the view to be displayed should de- 
pend on the position of the operator in relation to the object, 
10 which is the case for handheld display members and display 
members worn by the operator. This gives the operator the flexi- 
bility to move around the real object and immediately see the 
result at the real object of what he has been teaching. 

15 In the following, a specification of the information to be present- 
ed to the operator is presented in connection with a paint appli- 
cation. The graphical representation of the simulated result of 
the process is displayed normal to the pointing direction of the 
tool along the path, i.e. normal to a TCP. For example, paint 

20 strokes should be rendered perpendicular to the longitudinal di- 
rection of the brush along the path. 

Graphics that are obscured by a foreign object, or the object it- 
self, should not be rendered or rendered in a different way for 

25 example by using different color or by using transparent graph- 
ics. Alternatively, the graphics are not rendered. A method ac- 
cording to the invention comprises determining whether a part or 
parts of the generated graphics are hidden and generating said 
graphical representation based on whether it is hidden or not. 

30 Whenever a computer generated graphical element is hidden by 
the object or some other object, e.g. the operator's hand or the 
pointing member, the overlaid computer generated graphics 
should not be rendered, or rendered in a different way, so that 
the operator can differentiate the graphical hierarchy. 
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The graphics generator is adapted for generating different 
graphical representations depending on whether the result of the 
process is displayed from a front side or a backside in relation 
to the object. For example, rendered paint strokes should have a 
5 front side and a backside, thereby making it easier for the op- 
erator to distinguish between back and front. The user may 
choose to visualize the paint strokes with the actual simulated 
paint width or the paint strokes may be visualized as a single- 
colored line segment. The width of the path visualizes, for ex- 
10 ample, the paint stroke when the paint is on. 

For each waypoint specified by the operator, the specified brush 
number is displayed. A list of brushes is defined prior to the 
programming procedure and consists typically of three or four 

15 parameters that must be tuned to each other. Typical parame- 
ters are fan width, paint flow, atomizing airflow, shape airflow, 
high-voltage level, paint type, and paint color. The final goal is 
to have a brush with a defined width giving a uniform paint 
thickness within maximum and minimum values at a given paint 

20 speed. At least one specific brush may be defined as "paint-off", 
where paint flow, fan width, and airflow are all "0", alternatively 
paint flow is "0", while the other brush parameters are not "0". 

In one embodiment of the invention, a graphical representation 
25 of the relevant tool is shown moving through the generated robot 
path in real time. The relevant process information is overlaid 
the robot path, where this is specified in the program, so that 
the operator can verify the resulting robot program in real time. 
The speed of the display is defined in the graphic settings and 
30 may be altered during the actual display. The operator can spec- 
ify how fast this simulation should be performed in relation to 
real time. For example, the operator can choose to display the 
simulation in slow motion. It is possible to stop, pause, play 
backwards, and restart the display of the simulation. 
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It is possible to configure how graphics are presented on the 
graphical display member. For instance, the operator may 
choose to select or de-select one or more of the graphical de- 
scriptions. 

5 

Fig. 3 is an example of a view shown on the display. The view 
comprises the object 3 and a graphical representation of the re- 
corded waypoints 30, 32 and the path between the waypoints, 
projected on the object and path information. Each waypoint 30, 

10 32 taught by the operator is displayed as a dot. Each dot is pro- 
vided with process information related to that waypoint, such as 
the brush number. If the displayed brush number is "0" painting 
is off. The simulated robot path 34 is displayed as a line be- 
tween the waypoints 30, 32. Note that the generated path is not 

15 straight line between the waypoints. The robot path simulator 
generates the robot path, so that the tool passes through the 
waypoints at required speed and direction and then a smooth 
curve is to prefer before an angle. 

20 For each waypoint specified by the operator, information regard- 
ing the orientation of the tool is displayed. This is indicated by a 
small octagonal coordinate system showing the x, y, and z direc- 
tions. Preferably, each direction has a specific color code. The 
graphic generator also generates graphics representing the di- 

25 rection of the robot movement by adding arrows to the robot 
path shown. If the paint is on between two neighboring way- 
points 32, 36, the paint stroke is visualized between the two 
waypoints. The view shows a simulation of the resulting robot 
path and the paint result. 

30 

Figures 4a and 4b are flow chart illustrations of the method and 
the computer program product according to an embodiment of 
the present invention. It will be understood that each block of 
the flow chart can be implemented by computer program instruc- 
35 tions. The system can take three operational modes: a pro- 
gramming mode, a simulation mode, and an edit mode. The op- 
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erator chooses the system mode. In the programming mode, the 
user specifies the waypoints and input information about the 
process. In the simulation mode the robot path and/or the proc- 
ess are simulated and displayed to the operator. After the way 
5 points has been specified by the operator, alternatively after the 
robot program has been loaded into the computer's memory 
from the storage device, the user may set the system in edit 
mode. In edit mode, it is possible to remove stored waypoints, 
change the position of stored waypoints, change the orientation 
10 of stored waypoints, and change process related data, such as 
brush number of stored waypoints. 

Figure 4a shows the method steps when the system is in the 
programming mode. In block 30, a continuous stream of video 

15 signals is received from the camera unit 8. The image obtained 
comprises the object and the markers. In block 32, the system 
receives a record signal from the operator and an image is cap- 
tured for use by the image recognition. Process information 
specified by the operator is received and stored. In block 34, the 

20 markers are recognized by the image recognition, and the posi- 
tion and orientation of the pointing member are calculated in re- 
lation to the location of the object and stored. When the opera- 
tor has recorded the waypoints, the point generator generates a 
list of points on the path, block 36. In block 38 it is checked 

25 whether the points of the list are reachable, i.e. within the work- 
ing range of the robot and not singularities. If a point is outside 
the working range or is a singularity, the operator is notified, 
block 40, 42. When the operator has recorded the waypoints, he 
may change to the simulation mood. 

30 

Figure 4b shows the method when the simulation mode is se- 
lected. Before the simulation is started, the operator has to spe- 
cify what he wants to be simulated. He could choose either to 
simulate only the robot path or simulate both the robot path and 
35 the process. In block 50, the simulation is performed based on 
the list of waypoints with process specific information generated 
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during the programming mode and a model of the robot, a model 
of the robot controller and a model of the process. The simula- 
tion depends on the choice of the operator about what he wants 
to view. A graphical representation is generated of the robot 
5 path and/or the result of the process based on the simulation, 
block 52. As seen in block 54, the received video signal is com- 
bined with the generated graphics including registering the gen- 
erated graphical representation to the image of the object to 
provide a composed augmented reality image. A view of the 
10 combined video signal and graphics is displayed, block 56. The 
generated graphics shows, for example, whether processing is 
on or off. The display visualizes the view of the camera com- 
bined with the generated graphics. 

15 The software used for implementing the method according to the 
invention is partly based on software known to those skilled in 
the art. For example, the position and orientation may be gener- 
ated in ARToolKit based on the position and orientation of the point- 
ing marker in relation to the reference marker. The ARToolKit is 

20 developed by Washington University and the University of Hiro- 
shima and is an open-source software library that enables aug- 
mented reality applications to be built using accurate computer 
vision-based tracking techniques. For the application interface, 
the software Open GL may be used. OpenGL provides a library 

25 of 2D and 3D functions including modeling alterations, color, 
light and shade functions. Microsoft Vision SDK is a library for 
writing programs to perform image manipulation and analyses 
on computers. The augmented reality software includes algo- 
rithms for drawing graphics, such as points and lines, transfer- 

30 ring positions and orientations between different coordinate sys- 
tems, extracting and generating a sequence list of positions and 
orientations, capturing process related information, and drawing 
advanced graphics, such as color-coded points and lines repre- 
senting paint strokes with different widths and orientations. 
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The method according to invention is an off-line programming 
method, i.e. the method may be used outside the robot and the 
robot controller 

5 The present invention is not limited to the embodiments dis- 
closed but may be varied and modified within the scope of the 
following claims. For example, the present system is not limited 
to augmented reality systems utilizing a composed video image 
but may also be utilized in see-through augmented reality sys- 

10 terns, in which only the computer generated graphics are pre- 
sented to the user who views the graphics on a see-through lens 
with the real world in the background. The only difference be- 
tween the two systems is the translation of the camera coordi- 
nates into eye coordinates to compensate for the change in perspec- 

15 tive between the camera images and the actual real-world perspec- 
tive of the user. In either case, computer generated graphics are reg- 
istered to objects in the real world. In the see-through case, the 
computer-generated graphic is combined with the real-world object 
by the user rather than in the creation of a composed video image. 

20 

Alternatively, the waypoints are loaded into the system from an 
external system, e.g. a CAD application, or from a database of 
stored waypoints and process information. 

25 In one embodiment the system comprises a handheld aug- 
mented reality display device. The operator holds the handheld 
display device, showing the real world combined with overlaid 
computer-generated graphics. The computer-generated graphics 
may represent process related information, e.g. the programmed 

30 robot path, process specific information, target points and 
events. Either the operator's movements of his hand with a ges- 
ture-based recognition system or a pointing and interaction de- 
vice is used to specify the process related information to the 
system. 
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The computer-generated information represents operator input 
for robot programming. The operator utilizes the handheld AR 
display to see the result of what he is doing while he is making a 
new robot program. The handheld display has a camera inte- 
5 grated with the same field of view as the display. The camera 
captures live video of the real world. The system combines and 
synchronizes the live video with computer-generated graphics 
representing the operator's input and presents it on the hand- 
held AR display. The operator can freely move the handheld AR 
10 display in the environment, allowing the operator to see the local 
environment including the object from different viewpoints with 
overlaid computer generated graphics. The operator "looks" 
through the handheld AR display to see the generated robot pro- 
gram in relation to the real object. 

15 

A system according to this embodiment is shown in figure 5. The 
system comprises a handheld interaction/pointing device 1 with 
a tracking system for determining its position and orientation in 
relation to a world coordinate system 60. Alternatively the sys- 

20 tern comprises a gesture-based recognition system comprising a 
recognition system for recognizing and determining the position 
and orientation of a hand or fingers in relation to the world coor- 
dinate system. The system further comprises a handheld display 
device 62, e.g. a tablet PC or a PDA (Personal Digital Assis- 

25 tant). The display device comprises a display member 64, a 
camera 8 for capturing a stream of images of the environment 
mounted on or integrated with the display device 64. The cam- 
era 8 will be mounted in a fixed position in relation to the display 
device 64. The display device will be located along the camera 

30 view axis and at the camera's image plane. 

The system further comprises a tracking system 66 for determin- 
ing the position and orientation of the display devices. The 
tracking system is preferable mounted on the display device. 
35 The system also comprises a system for generating an augment- 
ed reality representation of computer-generated graphical infor- 
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mation overlaid the real world representation. The operator 2 
movements are sensed through the display tracking system 66. 

A wearable computer 68 contains the necessary software in or- 
5 der to generate the augmented reality environment based upon 
the video stream from the camera 8 and the computer-generated 
graphics. The wearable computer 68 also contains the neces- 
sary software needed to perform the desired task or process, 
e.g. generating a robot program and performing reachability 

10 check. Further, the wearable computer will generate the graph- 
ics, which will provide the augmented reality view. Finally, the 
wearable computer will contain a storage media in order to save, 
and restore previously saved, information. Communication of the 
information from the interaction/pointing device 1 to the wear- 

15 able computer 68 is done through a wired or wireless link. The 
operator carries the display device 62 which will provide him/her 
with an augmented reality view of the environment. The display 
device is of "video see through" type. 

20 Video see-through is used to generate and present an aug- 
mented reality world at the handheld AR display device. The 
camera integrated with the display device is used to capture a 
live video stream of the real world. The camera is located in re- 
lation with the display in such a way that it provides the same 

25 view, as the user would get by looking "through" the display de- 
vice. The live video stream combined with computer-generated 
graphics is presented in real-time at the display device. Addi- 
tional functionality includes camera zooming with output of the 
actual camera focal length. This will enable the system to dis- 

30 play the computer-generated graphics correctly while zooming. 
The camera may also be used for vision-based tracking if vision- 
based tracking is used as the tracking system. 



